(19) 



J 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 



(12) 



(43) Date of publication: 

29.12.1997 Bulletin 1997/52 

(21) Application number: 97304268.2 

(22) Date of filing: 18.06.1997 



01) EP 0 814 429 A2 

EUROPEAN PATENT APPLICATION 

(51) mt CIA G06T3/40 



(84) Designated Contracting States: 


(72) 


Inventor: Jones, Simon 


AT BE CH DE DK ES Fl FR GB GR IE IT LI LU MC 




Hungerford, Berkshire RG176XZ(GB) 


NL PT SE 








(74) 


Representative: Whitten, George Alan et al 


(30) Priority: 19.06.1996 GB 9612772 




R.G.C. Jenkins & Co., 






26 Caxton Street 


(71) Applicant: QUANTEL LIMITED 




London SW1H0RJ (GB) 


Newbury, Berkshire RG14 2LT (GB) 







(54) An image processing system 

(57) An image processing system 30 comprises a 
source image store 31, a destination image store 39, a 
calculation circuit 33, and a look-up table 45. Initial im- 
age data from the source store 31 is input to the calcu- 
lation circuit 33 together with interpolation parameter 
data from the look-up table 45. The calculation circuit 



33 interpolates the image data from the source store to 
produce data representing a magnified version of the 
image, which magnified image data is stored in the des- 
tination store 3a. The generation of unwanted artifacts 
in the image is avoided by using an interpolation function 
selected to provide a continuous transition between ad- 
jacent pixel values in the processed image data. 
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Description 

The invention relates to an image processing sys- 
tem. 

Image processing systems in which the painting or s 
drawing of a colour image can be simulated, or a portion 
of one image can be merged into another by electronic 
means are well known. One such graphic system is de- 
scribed in our British patent number 2,089,625 and cor- 
responding US patent number 4,51 4,81 8, the teachings 
of which are incorporated herein by reference. This sys- 
tem includes a user operable input device which may 
be used by the user to select from a range of colours 
and a range of intensities and to choose from a set of 
notional drawing implements for use in the painting or 
drawing. 

When a colour is chosen by the user, values repre- 
senting the components of the selected colour are 
stored in a colour register. An implement is chosen by 
selecting from among different implement representa- 
tions displayed on a display screen and the selected im- 
plement is defined by parameters conforming to a 3-di- 
mensional surface representing the profile of the imple- 
ment. Generally speaking the implement profile will 
have a high centre falling away to a lower value toward 
the periphery of the profile although other profiles may, 
of course, be defined. The implement profile represents 
the notional distribution of colour which would be applied 
by the implement to the image over the area of the image 
which it covers. 

The user operable input device is preferably a touch 
tablet and stylus combination. The touch tablet is ar- 
ranged to generate position signals designating the po- 
sition of the stylus relative to the touch tablet when the 
stylus is brought into proximity. When the stylus is ap- 
plied to the touch tablet a pressure signal representing 
the pressure applied via the stylus to the touch tablet is 
output from the stylus and stored in a pressure signal 
register. For some implements, representing say paint 
brushes, position signals are allowed to be generated 
for each movement of the stylus by the distance be- 
tween picture points or similar distance, whilst for other 
implements, say air brushes, position signals are gen- 
erated at regular time intervals, even if the stylus is held 
stationary on the touch tablet. 

When a position signal is produced, new video sig- 
nals (pixels) are derived for every picture point in the 
patch covered by the selected implement. An image 
store is provided and each new pixel is written at the 
appropriate picture point in the store. Such new pixels 
are derived bv a processing circuit in accordance with 
the selected colour data and the distribution of the se- 
lected implement, and in response also to the pressure 
applied to the stylus and to the value of the pixel previ- 
ously stored at the respective picture point in the store. 

The user, who it is envisaged would normally be an 
artist lacking experience in the use of computer based 
systems, paints or draws by choosing a desire colour 



and implement and then manipulating the stylus, caus- 
ing the touch tablet to generate a series of position sig- 
nals which define the path or positioning of the stylus. 
The processing circuit reads pixels from the image store 
for a patch of picture points in response to each position 
signal and these pixels are blended by the processor 
with signals representing the chosen colour in propor- 
tions depending upon the respective values of the brush 
profile and pressure. The blend is then written back to 
the picture store replacing the pixels previously stored 
therein. 

In general, the blending process is carried out a 
number of times for each picture point in the image store 
whether the implement is moving or stationary (assum- 
ing in the case of the moving implement that the patch 
covered by the implement is larger than the spacing be- 
tween picture points). The final proportion will depend 
on the number of processing operations performed per 
pixel. 

To enable the user to observe his creation, the 
stored picture is read repeatedly and the pixels are ap- 
plied to a TV-type colour monitor, so that the build-up of 
the picture can be observed. Of course such systems 
are not limited to TV-type formats and any suitable video 
format may be adopted. The system described avoids 
the problem of jagged edges in the image, an unpleas- 
ant stepping appearance normally associated with lines 
not lying horizontally or vertically in a raster display. 

Such systems are used in the processing of images 
from a wide variety of different sources, including stand- 
ard television images, high definition television (HDTV) 
images, movie film quality images and print quality im- 
ages. Thus, the resolution of the images varies widely 
from the 525 lines resolution of an NTSC television im- 
age, through the approximately 2000 x 3000 pixels def- 
inition of a movie image to the 10,000 x 8,000 or more 
pixels of a print quality image. Although the processing 
performed is essentially the same as that described in 
our above-mentioned patents regardless of the size or 
resolution of the image it is nevertheless necessary to 
approach the processing of a high resolution image dif- 
ferently than a low resolution image. 

For example when such a system is required to ef- 
fect painting on a high resolution graphic image, e.g. 
10,000 x 8,000 pixels, the largest normally available 
brush size which is limited to 200 pixels diameter ap- 
pears very small in relation to the overall image and the 
painting of large areas of the image is therefore a slow 
process. One way of dealing with this problem would be 
to provide a brush store with much larger brush stamp 
sizes but that solution would involve having a brush 
store of greatly increased capacity which would prove 
expensive. 

This problem may be dealt with by way of the tech- 
nique described in our British Patent Application pub- 
lished as GB-A-2,235,856 and corresponding US Pat- 
ent Publication US-A-5,1 42,616, the teachings of which 
are incorporated herein by reference. As is described in 
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these publications, the modification of a high resolution 
image is controlled using a low resolution stencil. During 
manipulation of the image a low resolution representa- 
tion is displayed on a monitor and manipulated under 
the control of a user defined low resolution stencil. Once 5 
the desired effect has been achieved, the low resolution 
stencil is up converted to high resolution and used to 
control the corresponding modification of the high reso- 
lution image. 

Thus, the problem is usually dealt with by way of a 
mode of operation as shown in Figure 1 of the accom- 
panying drawings. In this mode of operation, during 
"painting 0 the date representing the resolution version 
of the image (a) is processing to produce data repre- 
senting reduced resolution version of the image (b) 
which is displayed on a monitor (c). Suitably the image 
(a) is reduced by a factor of eight in both the horizontal 
and vertical directions to provide the image (b) of ap- 
proximately 1250 x 1000 pixels so that the image can 
be displayed on an HDTV monitor which is of compara- 
ble resolution. The user "paints" over this image using 
the brush sizes which are normally available (i.e. up to 
200 pixels in diameter) but the system actually "paints" 
data representing a stencil (d) into a stencil store. This 
stencil data defines a stencil having substantially the 
same resolution as the displayed image. The brush 
stamps which make up the brush stroke in the stencil 

(d) thus appear much larger in relation to the displayed 
image (c) than they do in relation to the high resolution 
image (a). When "painting" is complete, the data defin- 
ing the reduced resolution displayed image (b), which 
was only need for display purposes, is discarded leaving 
the data defining painted stencil. The stencil data is 
magnified up to the size of the high resolution image 
thereby producing data defining a high resolution stencil 

(e) . The high resolution stencil is then keyed over the 
high resolution image (a) by processing the data repre- 
senting the two images (a) and (e) to produce a high 
resolution image (f) comprising the subject of the origi- 
nal image (a) modified by the brush stroke defined by 
the stencil (c). 

Figure 2 of the accompanying drawings is a graph 
representing a section through a single patch of picture 
points "drawn" into a store in the manner described 
hereinabove. The single patch has a profile 1 corre- 
sponding to that defined by the implement data and thus 
the profile has aflat central region 2 in which pixels have 
a maximum value and side regions 3a, 3b where the 
pixel values fall away to zero. The patch is representa- 
tive of the stencil data that is created when the user 
"paints" into the image. 

When the stencil is enlarged the data is processed 
by a sizing/effects circuit within the system similar to that 
described in our co-pending British Patent Application 
published as GB-A-2, 305,052 and corresponding US 
Application No. 08/708,779, the teachings of which are 
incorporated herein by reference. The sizing circuit is 
arranged to enlarge or magnify the stencil by interpolat- 



ing the pixel values 4 to 9 to determine values at posi- 
tions between the pixels. Figure 3 of the accompanying 
drawings shows a portion 11 of the profile at the pixel 
points 4 to 9 inclusive enlarged bv a factor of 4. Thus, 
between adjacent initial pixel values three further inter- 
mediate pixel values 12, 13, 14 and 15, 16, 17 for ex- 
ample are generated by the sizing circuit by interpola- 
tion. 

Figures 2 and 3 represent pixel values along a sin- 
gle line, i.e. one-dimensional, in the stencil, but of 
course the stencil is in fact two-dimensional. The usual 
interpolation method employed to create the intermedi- 
ate pixel values 12 to 17 is bi-linear four-point interpo- 
lation in which intermediate pixel values in a two-dimen- 
sional image are derived from four adjacent pixels. As 
can be seen in Figure 3 this method produces interme- 
diate pixel values along a straight line between adjacent 
initial pixel values in the stencil. The resulting plot is 
therefore a piecewise linear approximation of the high 
resolution version of the initial lower resolution stencil. 
In many situations this is an acceptable approximation 
to the true high resolution version of the stencil. A prob- 
lem does, however, remain because the high resolution 
profile 1 1 is not smooth and continuous. The plot is linear 
between initial pixel points but changes abruptly at the 
initial pixel points. Consequently, the rate of change of 
the profile at one side of an initial pixel point is different 
than that at the other side of the initial pixel point. 

The human eye is able to perceive abrupt changes 
in the rate of change of intensity (or colour) which is the 
parameter represented by the pixel values of the graphs 
in Figures 2 and 3. Thus, as shown in Figure 4 of the 
accompanying drawings, the abrupt changes result in a 
patch 20 being seen as a series of bands 21 , 22, 23 of 
varying intensity with the boundary between one band 
and the next occurring at positions 24, 25, 26 corre- 
sponding to the positions of the initial sampling points 2 
to 7. These boundaries are known as second order ar- 
tifacts because they are artifacts which occur as a result 
of an abrupt change in the rate of change of an image 
parameter, such as the changes in the piecewise linear 
approximation of the sampling points 4 to 9. In contrast, 
a first order artifact is one which occurs as a result of an 
abrupt change of a parameter, say a sudden change 
from one intensity value to another. Second order arti- 
facts are clearly undesirable because they can be seen 
in images produced by the system. Second order arti- 
facts are unacceptable especially in still images such as 
photographs but also in moving images such as in tele- 
vision or movie film. 

The invention aims to provide an image processing 
system which is able to effect magnification of an image 
represented by digital data without producing second or- 
der artifacts. 

According to one aspect of the invention there is 
provided an image processing method for processing 
data defining a multiplicity of pixels that together form 
an initial image to produce processed data defining a 
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multiplicity of pixels that together form a magnified ver- 
sion of the image, in which method the data is processed 
to produce data representing one or more pixels be- 
tween pixels in the initial image by way of a non-linear 
interpolating function selected to minimise second order 
artifacts in the magnified version of the image. 

In another aspect of the invention there is provided 
an image processing apparatus for magnifying an image 
formed by an array of pixels each comprising digital data 
by providing, by interpolation between adjacent pixels 
of said image, pixels of a magnified form thereof, com- 
prising a memory in the form of a look-up table stored 
with filter coefficients representative of spline weighting 
functions and a calculation circuit adapted to employ 
said coefficients to calculate interpolated pixel values in 
the magnified image such that at least second order ar- 
tifacts in the magnified image are diminished. 

The invention also provides an image processing 
system for processing data representing an image to 
produce data representing a largerversion of the image, 
in which system initial image data defining a multiplicity 
of pixel values predetermined positions in an initial im- 
age is processed to produce data representing addition- 
al pixel values at positions between the predetermined 
positions, which processing is effected by way of an in- 
terpolation function selected to provide a continuous 
transition between adjacent pixel values in the proc- 
essed image data. 

The above and further features of the invention are 
set forth with particularity in the appended claims and 
together with advantages thereof will become clearer 
from consideration of the following detailed description 
of an exemplary embodiment of the invention given with 
reference to the accompanying drawings. 

In the drawings: 

Figure 1 illustrates a known procedure for applying 
a brush stroke to an image, as already described 
Therein; 

Figure 2 illustrates a section through a single patch 
of pixels in an image, as already described herein; 
Figure 3 illustrates a magnified version of a portion 
of the patch in Figure 2, as already described here- 
in; 

Figure 4 illustrates artifacts in the magnified portion 
of the patch; 

Figure 5 is a schematic functional block diagram of 
an image processing system embodying the inven- 
tion; and 

Figure 6 shows in more detail a calculation circuit 
of the Figure 5 system. 

Turning now to Figure 5 of the accompanying draw- 
ings there is shown an image processing system 30 
which is essentially the same as the system described 
in the abovementioned GB-A-2,305,052. As shown in 
Figure 5, the system 30 comprises a source image store 
31 for storing image data from any suitable external 



source including a video camera, a video tape recorder, 
a video disc store or an image scanner for example. The 
data in the source image store 30 defines a multiplicity 
of pixels that together form an image (for example a 
5 stencil) to be magnified by the system 30. The source 
image store 31 is connected to a calculation circuit 33 
which received pixel data from the store 31 by way of a 
data bus 35. The calculation circuit is arranged to per- 
form a magnifying effect on the data it receives from the 
10 store 31 by the creation of interpolated pixel data. The 
magnified data is output on data bus 37 from the calcu- 
lation circuit 33 to a destination image store 39. A source 
address generator 41 determines the pixels and the or- 
der thereof to be supplied from the source store 31 to 
is the calculation circuit 33, and a destination address gen- 
erator 43 determines the address in the destination 
store to which the magnified image pixel data from the 
calculation mechanism is to be directed. 

In addition to the source pixel data received from 
20 the source store 31, the calculating circuit 33 also re- 
ceives weighting coefficients pertaining to one of a 
number of spline functions which, preferably, are B- 
spline functions which are applied to the source pixels 
supplied to the calculating circuit to provide, as herein- 
25 after described in further detail, the output pixels sup- 
plied to the destination store 39. Although B-spline func- 
tions are those, of choice, preferably employed in the 
system it is pointed out that any non-linear interpolation 
function which has the effect of diminishing the visibility 
30 in the destination image of second order artifacts may 
be employed. That is, any function that does not result 
in the creation of abrupt changes in the rate of change 
of an image parameter such as intensity or colour, but 
instead produces a smooth local transition at the initial 
35 pixel point, may be used. Such functions are perse well 
known and need not be described in detail herein. The 
use of splines in image data interpolation is discussed 
in detail in "Computer Graphics - Principles and Prac- 
tice 0 by Voley, van Dam, et al. 2nd ed. published 1990 
40 by Addison-Wesley, for example. 

The spline weighting function coefficients are 
stored in a memory comprising a look-up table (LUT) 45 
and the coefficients of the spline employed by the cal- 
culator circuit 33 are selected by a magnification and 
45 interpolation controller 37 which also instructs the 
source and destination address generators 41 and 43 
respectively to generate the pixel addresses which are 
required for the magnification calculation and the pixels 
addresses at which the pixels of the magnified image 
50 are to be stored. 

In its turn the magnification and interpolation con- 
troller 47 is under the control of a top level controller 49 
which instructs the controller 47 as the type, e.g. B- 
spline, and the degree of magnification operation to be 
55 effected. 

Figure 6 of the accompanying drawings shows the 
calculation circuit 33 in greater detail. As shown in Fig- 
ure 6, the calculation circuit 33 comprises four multipli- 
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ers 50, 51 , 52, 53 and an adder 54 for adding together 
the outputs from the multipliers. The four multipliers 50, 
51 , 52, 53 receive respective pixel data P A to P 4 on the 
bus 35 from the image store 31 and receive respective 
coefficient data a^ and a 4 from the coefficient look up 
table 45. 

The pixel data to P 4 and the coefficients a^ to 84 
are operated on by the multipliers 50 to 53 and the adder 
54 to produce an intermediate result R = a-jP-j + a 2 P 2 + 
a 3 P 3 + a 4 P 4 . This intermediate result R is stored in the 
register 55 and from there is output to the bus 33 for 
storage in the destination image store 39. 

This calculation is repeated for each of the interme- 
diate pixels between the initial pixel values 2 to 7 (see 
Figures 1 and 2). Depending on the scale of the magni- 
fication the initial pixel values 2 to 7 may be output to 
the destination store unaltered or they may simply be 
replaced by pixel values at intermediate positions adja- 
cent the initial positions. A more detailed description of 
the calculation circuit is given in the abovementioned 
GB-A-2,305,052 although it should be noted that as de- 
scribed therein the calculation circuit is configured to op- 
erate as a filter rather than as an interpolator. 

Brightness, Y, and colour difference U and V values 
of the destination image pixels are separately calculated 
in the manner described to provide the total data for 
each destination image pixel. The destination image can 
be sent to be displayed on a display monitor, stored in 
a disc store or on a tape or printed by a printer. 

Although a look-up table 45 has been referred to for 
storage of the spline coefficients, the coefficients could 
be calculated "on the fly" by the magnification and inter- 
polation controller and supplied directly to the calcula- 
tion circuit, the use of B-spline weighting functions is 
preferred as they effect local smoothing of the brush pro- 
file curve by calculation of each interpolated pixel using 
a few, suitably about three, successive pixel values. 

Having thus described the present invention by ref- 
erence to a preferred embodiment it is to be well under- 
stood that the embodiment in question is exemplary only 
and that modifications and variations such as will occur 
to those possessed of appropriate knowledge and skills 
may be made without departure from the spirit and 
scope of the invention as set forth in the appended 
claims and equivalents thereof. 



Claims 

1. An image processing method for processing data 
defining a multiplicity of pixels that together form an 
initial image to produce processed data defining a 
multiplicity of pixels that together form a magnified 
version of the image, in which method the data is 
processed to produce data representing one or 
more pixels between pixels in the initial image by 
way of a non-linear interpolating function selected 
to minimise second order artifacts in the magnified 



version of the image. 

2. An image processing method as claimed in claim 1 , 
wherein said non-linear interpolation functions is a 

5 spline function. 

3. An image processing method as claimed in claim 2, 
wherein said spline function is a local control spline 
function. 

70 

4. An image processing method as claimed in claim 3, 
wherein said local control spline function is a B- 
spline function. 

75 5. An image processing apparatus for magnifying an 
image formed by an array of pixels each comprising 
digital data by providing, by interpolation between 
adjacent pixels of said image, pixels of a magnified 
form thereof, comprising a memory in the form of a 
20 look-up table stored with filter coefficients repre- 
sentative of spline weighting functions and a calcu- 
lation circuit adapted to employ said coefficients to 
calculate interpolated pixel values in the magnified 
image such that at least second order artifacts in 
25 the magnified image are diminished. 

6. An image processing apparatus as claimed in claim 

5, further comprising a magnification and interpola- 
tion controller for selecting the spline weighting 

30 function coefficients to be employed by said calcu- 
lation means. 

7. An image processing apparatus as claimed in claim 

6, further comprising a source store for storing data 
35 defining said image, a destination store for storing 

data defining said magnified image, a source ad- 
dress generator, and a destination address gener- 
ator, and wherein said magnification and interpola- 
tion controller is adapted to instruct the address 
40 generators, in the case of the source address gen- 
erator to locate image pixels required for calculation 
of a corresponding magnified image pixel and in the 
case of the destination image address generator to 
calculate the address of said corresponding magni- 
45 fied image pixel. 

8. An image processing apparatus as claimed in claim 
6 or 7, further comprising a look-up table for storing 
said spline weighting function coefficients. 

50 

9. An image processing apparatus as claimed in claim 
8, wherein said look-up table is adapted to store a 
plurality of spline weighting functions and said mag- 
nification and interpolation control servers to select 

55 the function coefficients of one of said functions for 
employment by the calculation circuit. 

10. An image processing apparatus as claimed in any 
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one of claims 5 to 9, wherein said spline weighting 
functions are B-spline, local control functions. 

11. An image processing apparatus as claimed in any 
one of claims 5 to 10, wherein said calculation s 
means comprises a finite impulse response filter. 

12. An image processing system for processing data 
representing an image to produce data represent- 
ing a larger version of the image, in which system '0 
initial image data defining a multiplicity of pixel val- 
ues predetermined positions in an initial image is 
processed to produce data representing additional 
pixel values at positions between the predeter- 
mined positions, which processing is effected by is 
way of an interpolation function selected to provide 

a continuous transition between adjacent pixel val- 
ues in the processed image data. 
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(54) An image processing system 



(57) An image processing system 30 comprises a 
source image store 31 , a destination image store 39, a 
calculation circuit 33, and a look-up table 45. Initial im- 
age data from the source store 31 is input to the calcu- 
lation circuit 33 together with interpolation parameter 
data from the look-up table 45. The calculation circuit 



33 interpolates the image data from the source store to 
produce data representing a magnified version of the 
image, which magnified image data is stored in the des- 
tination store 3a. The generation of unwanted artifacts 
in the image is avoided by using an interpolation function 
selected to provide a continuous transition between ad- 
jacent pixel values in the processed image data. 



3 °\ 



TOP LEVEL 
CONTROL 



Li 

/ 



-L9 



MAGNIFICATION 
& 

INTERPOLATION 
CONTROL 



SOURCE 
ADDRESS 
GEN. 



CM 

< 

O) 
CM 



00 

o 

Q. 
UJ 



31 



LUT 



SOURCE 
IMAGE STORE 



7 

35 



33 



39 



DEST. 
ADDRESS 
GEN. 



CALCULATION 
CIRCUIT 



DESTINATION 
IMAGE STORE 



FIG. 5 



37 



Printed by Jouve. 75001 PARIS (FR) 



1 



EP 0 814 429 A2 



2 



Description 

The invention relates to an image processing sys- 
tem. 

Image processing systems in which the painting or s 
drawing of a colour image can be simulated, or a portion 
of one image can be merged into another by electronic 
means are well known. One such graphic system is de- 
scribed in our British patent number 2,089,625 and cor- 
responding US patent number 4,51 4,81 8, the teachings 
of which are incorporated herein by reference. This sys- 
tem includes a user operable input device which may 
be used by the user to select from a range of colours 
and a range of intensities and to choose from a set of 
notional drawing implements for use in the painting or 
drawing. 

When a colour is chosen by the user, values repre- 
senting the components of the selected colour are 
stored in a colour register. An implement is chosen by 
selecting from among different implement representa- 
tions displayed on a display screen and the selected im- 
plement is defined by parameters conforming to a 3-di- 
mensional surface representing the profile of the imple- 
ment. Generally speaking the implement profile will 
have a high centre falling away to a lower value toward 
the periphery of the profile although other profiles may, 
of course, be defined. The implement profile represents 
the notional distribution of colour which would be applied 
by the implement to the image over the area of the image 
which it covers. 

The user operable input device is preferably a touch 
tablet and stylus combination. The touch tablet is ar- 
ranged to generate position signals designating the po- 
sition of the stylus relative to the touch tablet when the 
stylus is brought into proximity. When the stylus is ap- 
plied to the touch tablet a pressure signal representing 
the pressure applied via the stylus to the touch tablet is 
output from the stylus and stored in a pressure signal 
register. For some implements, representing say paint 
brushes, position signals are allowed to be generated 
for each movement of the stylus by the distance be- 
tween picture points or similar distance, whilst for other 
implements, say air brushes, position signals are gen- 
erated at regular time intervals, even if the stylus is held 
stationary on the touch tablet. 

When a position signal is produced, new video sig- 
nals (pixels) are derived for every picture point in the 
patch covered by the selected implement. An image 
store is provided and each new pixel is written at the 
appropriate picture point in the store. Such new pixels 
are derived bv a processing circuit in accordance with 
the selected colour data and the distribution of the se- 
lected implement, and in response also to the pressure 
applied to the stylus and to the value of the pixel previ- 
ously stored at the respective picture point in the store. 

The user, who it is envisaged would normally be an 
artist lacking experience in the use of computer based 
systems, paints or draws by choosing a desire colour 



and implement and then manipulating the stylus, caus- 
ing the touch tablet to generate a series of position sig- 
nals which define the path or positioning of the stylus. 
The processing circuit reads pixels from the image store 
for a patch of picture points in response to each position 
signal and these pixels are blended by the processor 
with signals representing the chosen colour in propor- 
tions depending upon the respective values of the brush 
profile and pressure. The blend is then written back to 
the picture store replacing the pixels previously stored 
therein. 

In general, the blending process is carried out a 
number of times for each picture point in the image store 
whether the implement is moving or stationary (assum- 
ing in the case of the moving implement that the patch 
covered by the implement is larger than the spacing be- 
tween picture points). The final proportion will depend 
on the number of processing operations performed per 
pixel. 

To enable the user to observe his creation, the 
stored picture is read repeatedly and the pixels are ap- 
plied to a TV-type colour monitor, so that the build-up of 
the picture can be observed. Of course such systems 
are not limited to TV-type formats and any suitable video 
format may be adopted. The system described avoids 
the problem of jagged edges in the image, an unpleas- 
ant stepping appearance normally associated with lines 
not lying horizontally or vertically in a raster display. 

Such systems are used in the processing of images 
from a wide variety of different sources, including stand- 
ard television images, high definition television (HDTV) 
images, movie film quality images and print quality im- 
ages. Thus, the resolution of the images varies widely 
from the 525 lines resolution of an NTSC television im- 
age, through the approximately 2000 x 3000 pixels def- 
inition of a movie image to the 10,000 x 8,000 or more 
pixels of a print quality image. Although the processing 
performed is essentially the same as that described in 
our above-mentioned patents regardless of the size or 
resolution of the image it is nevertheless necessary to 
approach the processing of a high resolution image dif- 
ferently than a low resolution image. 

For example when such a system is required to ef- 
fect painting on a high resolution graphic image, e.g. 
10,000 x 8,000 pixels, the largest normally available 
brush size which is limited to 200 pixels diameter ap- 
pears very small in relation to the overall image and the 
painting of large areas of the image is therefore a slow 
process. One way of dealing with this problem would be 
to provide a brush store with much larger brush stamp 
sizes but that solution would involve having a brush 
store of greatly increased capacity which would prove 
expensive. 

This problem may be dealt with by way of the tech- 
nique described in our British Patent Application pub- 
lished as GB-A-2,235,856 and corresponding US Pat- 
ent Publication US-A-5, 1 42,61 6, the teachings of which 
are incorporated herein by reference. As is described in 
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these publications, the modification ot a high resolution 
image is controlled using a low resolution stencil. During 
manipulation of the image a low resolution representa- 
tion is displayed on a monitor and manipulated under 
the control of a user defined low resolution stencil. Once 
the desired effect has been achieved, the low resolution 
stencil is up converted to high resolution and used to 
control the corresponding modification of the high reso- 
lution image. 

Thus, the problem is usually dealt with by way of a 
mode of operation as shown in Figure 1 of the accom- 
panying drawings. In this mode of operation, during 
"painting" the date representing the resolution version 
of the image (a) is processing to produce data repre- 
senting reduced resolution version of the image (b) 
which is displayed on a monitor (c). Suitably the image 
(a) is reduced by a factor of eight in both the horizontal 
and vertical directions to provide the image (b) of ap- 
proximately 1250 x 1000 pixels so that the image can 
be displayed on an HDTV monitor which is of compara- 
ble resolution. The user "paints" over this image using 
the brush sizes which are normally available (i.e. up to 
200 pixels in diameter) but the system actually "paints" 
data representing a stencil (d) into a stencil store. This 
stencil data defines a stencil having substantially the 
same resolution as the displayed image. The brush 
stamps which make up the brush stroke in the stencil 

(d) thus appear much larger in relation to the displayed 
image (c) than they do in relation to the high resolution 
image (a). When "painting" is complete, the data defin- 
ing the reduced resolution displayed image (b), which 
was only need for display purposes, is discarded leaving 
the data defining painted stencil. The stencil data is 
magnified up to the size of the high resolution image 
thereby producing data defining a high resolution stencil 

(e) . The high resolution stencil is then keyed over the 
high resolution image (a) by processing the data repre- 
senting the two images (a) and (e) to produce a high 
resolution image (f) comprising the subject of the origi- 
nal image (a) modified by the brush stroke defined by 
the stencil (c). 

Figure 2 of the accompanying drawings is a graph 
representing a section through a single patch of picture 
points "drawn" into a store in the manner described 
hereinabove. The single patch has a profile 1 corre- 
sponding to that defined by the implement data and thus 
the profile has aflat central region 2 in which pixels have 
a maximum value and side regions 3a, 3b where the 
pixel values fall away to zero. The patch is representa- 
tive of the stencil data that is created when the user 
"paints" into the image. 

When the stencil is enlarged the data is processed 
by a sizing/effects circuit within the system similar to that 
described in our co-pending British Patent Application 
published as GB-A-2,305,052 and corresponding US 
Application No. 08/708,779, the teachings of which are 
incorporated herein by reference. The sizing circuit is 
arranged to enlarge or magnify the stencil by interpolat- 



ing the pixel values 4 to 9 to determine values at posi- 
tions between the pixels. Figure 3 of the accompanying 
drawings shows a portion 11 of the profile at the pixel 
points 4 to 9 inclusive enlarged bv a factor of 4. Thus. 
s between adjacent initial pixel values three further inter- 
mediate pixel values 12, 13, 14 and 15, 16, 17 for ex- 
ample are generated by the sizing circuit by interpola- 
tion. 

Figures 2 and 3 represent pixel values along a sin- 
10 gie line, i.e. one-dimensional, in the stencil, but of 
course the stencil is in fact two-dimensional. The usual 
interpolation method employed to create the intermedi- 
ate pixel values 12 to 17 is bi-linear four-point interpo- 
lation in which intermediate pixel values in a two-dimen- 
i$ sional image are derived from four adjacent pixels. As 
can be seen in Figure 3 this method produces interme- 
diate pixel values along a straight line between adjacent 
initial pixel values in the stencil. The resulting plot is 
therefore a piecewise linear approximation of the high 
20 resolution version of the initial lower resolution stencil. 
In many situations this is an acceptable approximation 
to the true high resolution version of the stencil. A prob- 
lem does, however, remain because the high resolution 
profile 1 1 is not smooth and continuous. The plot is linear 
25 between initial pixel points but changes abruptly at the 
initial pixel points. Consequently, the rate of change of 
the profile at one side of an initial pixel point is different 
than that at the other side of the initial pixel point. 
The human eye is able to perceive abrupt changes 
30 in the rate of change of intensity (or colour) which is the 
parameter represented by the pixel values of the graphs 
in Figures 2 and 3. Thus, as shown in Figure 4 of the 
accompanying drawings, the abrupt changes result in a 
patch 20 being seen as a series of bands 21 , 22, 23 of 
35 varying intensity with the boundary between one band 
and the next occurring at positions 24, 25, 26 corre- 
sponding to the positions of the initial sampling points 2 
to 7. These boundaries are known as second order ar- 
tifacts because they are artifacts which occur as a result 
40 of an abrupt change in the rate of change of an image 
parameter, such as the changes in the piecewise linear 
approximation of the sampling points 4 to 9. In contrast, 
a first order artifact is one which occurs as a result of an 
abrupt change of a parameter, say a sudden change 
45 from one intensity value to another. Second order arti- 
facts are clearly undesirable because they can be seen 
in images produced by the system. Second order arti- 
facts are unacceptable especially in still images such as 
photographs but also in moving images such as in tele- 
50 vision or movie film. 

The invention aims to provide an image processing 
system which is able to effect magnification of an image 
represented by digital data without producing second or- 
der artifacts. 

55 According to one aspect of the invention there is 
provided an image processing method for processing 
data defining a multiplicity of pixels that together form 
an initial image to produce processed data defining a 
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multiplicity of pixels that together form a magnified ver- 
sion of the image, in which method the data is processed 
to produce data representing one or more pixels be- 
tween pixels in the initial image by way of a non-linear 
interpolating function selected to minimise second order s 
artifacts in the magnified version of the image. 

In another aspect of the invention there is provided 
an image processing apparatus for magnifying an image 
formed by an array of pixels each comprising digital data 
by providing, by interpolation between adjacent pixels 
of said image, pixels of a magnified form thereof, com- 
prising a memory in the form of a look-up table stored 
with filter coefficients representative of spline weighting 
functions and a calculation circuit adapted to employ 
said coefficients to calculate interpolated pixel values in 
the magnified image such that at least second order ar- 
tifacts in the magnified image are diminished. 

The invention also provides an image processing 
system for processing data representing an image to 
produce data representing a larger version of the image, 
in which system initial image data defining a multiplicity 
of pixel values predetermined positions in an initial im- 
age is processed to produce data representing addition- 
al pixel values at positions between the predetermined 
positions, which processing is effected by way of an in- 
terpolation function selected to provide a continuous 
transition between adjacent pixel values in the proc- 
essed image data. 

The above and further features of the invention are 
set forth with particularity in the appended claims and 
together with advantages thereof will become clearer 
from consideration of the following detailed description 
of an exemplary embodiment of the invention given with 
reference to the accompanying drawings. 

In the drawings: 

Figure 1 illustrates a known procedure for applying 
a brush stroke to an image, as already described 
Therein; 

Figure 2 illustrates a section through a single patch 
of pixels in an image, as already described herein; 
Figure 3 illustrates a magnified version of a portion 
of the patch in Figure 2, as already described here- 
in; 

Figure 4 illustrates artifacts in the magnified portion 
of the patch; 

Figure 5 is a schematic functional block diagram of 
an image processing system embodying the inven- 
tion; and 

Figure 6 shows in more detail a calculation circuit 
of the Figure 5 system. 

Turning now to Figure 5 of the accompanying draw- 
ings there is shown an image processing system 30 
which is essentially the same as the system described 
in the abovementioned GB-A-2,305,052. As shown in 
Figure 5, the system 30 comprises a source image store 
31 for storing image data from any suitable external 



source including a video camera, a video tape recorder, 
a video disc store or an image scanner for example. The 
data in the source image store 30 defines a multiplicity 
of pixels that together form an image (for example a 
stencil) to be magnified by the system 30. The source 
image store 31 is connected to a calculation circuit 33 
which received pixel data from the store 31 by way of a 
data bus 35. The calculation circuit is arranged to per- 
form a magnifying effect on the data it receives from the 
store 31 by the creation of interpolated pixel data. The 
magnified data is output on data bus 37 from the calcu- 
lation circuit 33 to a destination image store 39. A source 
address generator 41 determines the pixels and the or- 
der thereof to be supplied from the source store 31 to 
the calculation circuit 33, and a destination address gen- 
erator 43 determines the address in the destination 
store to which the magnified image pixel data from the 
calculation mechanism is to be directed. 

In addition to the source pixel data received from 
the source store 31 , the calculating circuit 33 also re- 
ceives weighting coefficients pertaining to one of a 
number of spline functions which, preferably, are B- 
spline functions which are applied to the source pixels 
supplied to the calculating circuit to provide, as herein- 
after described in further detail, the output pixels sup- 
plied to the destination store 39. Although B-spline func- 
tions are those, of choice, preferably employed in the 
system it is pointed out that any non-linear interpolation 
function which has the effect of diminishing the visibility 
in the destination image of second order artifacts may 
be employed. That is, any function that does not result 
in the creation of abrupt changes in the rate of change 
of an image parameter such as intensity or colour, but 
instead produces a smooth local transition at the initial 
pixel point, may be used. Such functions are perse well 
known and need not be described in detail herein. The 
use of splines in image data interpolation is discussed 
in detail in "Computer Graphics - Principles and Prac- 
tice" by Voley, van Dam, e\a\. 2nd ed. published 1990 
by Addison -Wesley, for example. 

The spline weighting function coefficients are 
stored in a memory comprising a look-up table (LUT) 45 
and the coefficients of the spline employed by the cal- 
culator circuit 33 are selected by a magnification and 
interpolation controller 37 which also instructs the 
source and destination address generators 41 and 43 
respectively to generate the pixel addresses which are 
required for the magnification calculation and the pixels 
addresses at which the pixels of the magnified image 
are to be stored. 

In its turn the magnification and interpolation con- 
troller 47 is under the control of a top level controller 49 
which instructs the controller 47 as the type, e.g. B- 
spline, and the degree of magnification operation to be 
effected. 

Figure 6 of the accompanying drawings shows the 
calculation circuit 33 in greater detail. As shown in Fig- 
ure 6, the calculation circuit 33 comprises four multipli- 
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ers 50, 51, 52, 53 and an adder 54 for adding together 
the outputs from the multipliers. The four multipliers 50, 
51 , 52, 53 receive respective pixel data P n to P 4 on the 
bus 35 from the image store 31 and receive respective 
coefficient data a 1 and a 4 from the coefficient look up 
table 45. 

The pixel data P 1 to P 4 and the coefficients to a 4 
are operated on by the multipliers 50 to 53 and the adder 
54 to produce an intermediate result R = a 1 P 1 + a 2 Pa + 
^3^3 + a A P A . This intermediate result R is stored in the 
register 55 and from there is output to the bus 33 for 
storage in the destination image store 39. 

This calculation is repeated for each of the interme- 
diate pixels between the initial pixel values 2 to 7 (see 
Figures 1 and 2). Depending on the scale of the magni- 
fication the initial pixel values 2 to 7 may be output to 
the destination store unaltered or they may simply be 
replaced by pixel values at intermediate positions adja- 
cent the initial positions. A more detailed description of 
the calculation circuit is given in the abovementioned 
GB-A-2,305,052 although it should be noted that as de- 
scribed therein the calculation circuit is configured to op- 
erate as a filter rather than as an interpolator. 

Brightness, Y, and colour difference U and V values 
of the destination image pixels are separately calculated 
in the manner described to provide the total data for 
each destination image pixel. The destination image can 
be sent to be displayed on a display monitor, stored in 
a disc store or on a tape or printed by a printer. 

Although a look-up table 45 has been referred to for 
storage of the spline coefficients, the coefficients could 
be calculated "on the fly" by the magnification and inter- 
polation controller and supplied directly to the calcula- 
tion circuit, the use of B-spline weighting functions is 
preferred as they effect local smoothing of the brush pro- 
file curve by calculation of each interpolated pixel using 
a few, suitably about three, successive pixel values. 

Having thus described the present invention by ref- 
erence to a preferred embodiment it is to be well under- 
stood that the embodiment in question is exemplary only 
and that modifications and variations such as will occur 
to those possessed of appropriate knowledge and skills 
may be made without departure from the spirit and 
scope of the invention as set forth in the appended 
claims and equivalents thereof. 



Claims 

1. An image processing method for processing data 
defining a multiplicity of pixels that together form an 
initial image to produce processed data defining a 
multiplicity of pixels that together form a magnified 
version of the image, in which method the data is 
processed to produce data representing one or 
more pixels between pixels in the initial image by 
way of a non-linear interpolating function selected 
to minimise second order artifacts in the magnified 



version of the image. 

2. An image processing method as claimed in claim 1 , 
wherein said non-linear interpolation functions is a 

5 spline function. 

3. An image processing method as claimed in claim 2, 
wherein said spline function is a local control spline 
function. 

10 

4. An image processing method as claimed in claim 3, 
wherein said local control spline function is a B- 
spline function. 

*5 5, An image processing apparatus for magnifying an 
image formed by an array of pixels each comprising 
digital data by providing, by interpolation between 
adjacent pixels of said image, pixels of a magnified 
form thereof, comprising a memory in the form of a 
20 look-up table stored with filter coefficients repre- 
sentative of spline weighting functions and a calcu- 
lation circuit adapted to employ said coefficients to 
calculate interpolated pixel values in the magnified 
image such that at least second order artifacts in 
25 the magnified image are diminished. 

6. An image processing apparatus as claimed in claim 

5, further comprising a magnification and interpola- 
tion controller for selecting the spline weighting 

30 function coefficients to be employed by said calcu- 
lation means. 

7. An image processing apparatus as claimed in claim 

6, further comprising a source store for storing data 
35 defining said image, a destination store for storing 

data defining said magnified image, a source ad- 
dress generator, and a destination address gener- 
ator, and wherein said magnification and interpola- 
tion controller is adapted to instruct the address 
40 generators, in the case of the source address gen- 
erator to locate image pixels required for calculation 
of a corresponding magnified image pixel and in the 
case of the destination image address generator to 
calculate the address of said corresponding magni- 
45 tied image pixel. 

8. An image processing apparatus as claimed in claim 
6 or 7, further comprising a look-up table for storing 
said spline weighting function coefficients. 

50 

9. An image processing apparatus as claimed in claim 
8, wherein said look-up table is adapted to store a 
plurality of spline weighting functions and said mag- 
nification and interpolation control servers to select 

55 the function coefficients of one of said functions for 
employment by the calculation circuit. 

10. An image processing apparatus as claimed in any 
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one of claims 5 to 9, wherein said spline weighting 
functions are B-spline, local control functions. 

11. An image processing apparatus as claimed in any 
one of claims 5 to 10, wherein said calculation 5 
means comprises a finite impulse response filter 

12. An image processing system for processing data 
representing an image to produce data represent- 
ing a larger version of the image, in which system io 
initial image data defining a multiplicity of pixel val- 
ues predetermined positions in an initial image is 
processed to produce data representing additional 
pixel values at positions between the predeter- 
mined positions, which processing is effected by is 
way of an interpolation function selected to provide 

a continuous transition between adjacent pixel val- 
ues in the processed image data. 
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Publication Title: 

An image processing system 



Abstract: 

An image processing system 30 comprises a source image store 31, a 
destination image store 39, a calculation circuit 33, and a look-up table 45. Initial 
image data from the source store 31 is input to the calculation circuit 33 together 
with interpolation parameter data from the look-up table 45. The calculation 
circuit 33 interpolates the image data from the source store to produce data 
representing a magnified version of the image, which magnified image data is 
stored in the destination store 3a. The generation of unwanted artifacts in the 
image is avoided by using an interpolation function selected to provide a 
continuous transition between adjacent pixel values in the processed image data. 
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